Concurrent program execution optimization

ABSTRACT

An architecture for a load-balanced groups of multi-stage manycore processors shared dynamically among a set of software applications, with capabilities for destination task defined intra-application prioritization of inter-task communications (ITC), for architecture-based ITC performance isolation between the applications, as well as for prioritizing application task instances for execution on cores of manycore processors based at least in part on which of the task instances have available for them the input data, such as ITC data, that they need for executing.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 17/195,174 filed Mar. 8, 2021, which is a continuation applicationof U.S. application Ser. No. 16/434,581 filed Jun. 7, 2019 (now U.S.Pat. No. 10,942,778), which is a continuation application of U.S.application Ser. No. 15/267,153 filed Sep. 16, 2016 (now U.S. Pat. No.10,318,353), which is a continuation application of U.S. applicationSer. No. 14/318,512 filed Jun. 27, 2014 (now U.S. Pat. No. 9,448,847),which claims the benefit and priority of the following provisionalapplications:

[1] U.S. Provisional Application No. 61/934,747 filed Feb. 1, 2014; and[2] U.S. Provisional Application No. 61/869,646 filed Aug. 23, 2013;

This application is also related to the following co-pending or patentedapplications:

[3] U.S. Utility application Ser. No. 13/184,028, filed Jul. 15, 2011;[4] U.S. Utility application Ser. No. 13/270,194, filed Oct. 10, 2011;[5] U.S. Utility application Ser. No. 13/277,739, filed Nov. 21, 2011;[6] U.S. Utility application Ser. No. 13/297,455, filed Nov. 16, 2011;[7] U.S. Utility application Ser. No. 13/684,473, filed Nov. 23, 2012;[8] U.S. Utility application Ser. No. 13/717,649, filed Dec. 17, 2012;[9] U.S. Utility application Ser. No. 13/901,566, filed May 24, 2013;and[10] U.S. Utility application Ser. No. 13/906,159, filed May 30, 2013.

All above identified applications are hereby incorporated by referencein their entireties for all purposes.

BACKGROUND Technical Field

This invention pertains to the field of information processing,particularly to techniques for managing execution of multipleconcurrent, multi-task software programs on parallel processinghardware.

Descriptions of the Related Art

Conventional microprocessor and computer system architectures rely onsystem software for handling runtime matters relating to sharingprocessing resources among multiple application programs and theirinstances, tasks etc., as well as orchestrating the concurrent (paralleland/or pipelined) execution between and within the individualapplications sharing the given set of processing resources. However, thesystem software consumes by itself ever increasing portions of thesystem processing capacity, as the number of applications, theirinstances and tasks and the pooled processing resources would grow, aswell as the more frequently the optimizations of the dynamic resourcemanagement among the applications and their tasks would be needed to beperformed, in response to variations in the applications' and theirinstances' and tasks' processing loads etc. variables of the processingenvironment. As such, the conventional approaches for supporting dynamicexecution of concurrent programs on shared processing capacity poolswill not scale well.

This presents significant challenges to the scalability of the networkedutility (‘cloud’) computing model, in particular as there will be acontinuously increasing need for greater degrees of concurrentprocessing also at intra-application levels, in order to enableincreasing individual application on-time processing throughputperformance, without the automatic speed-up from processor clock ratesbeing available due to the practical physical and economic constraintsfaced by the semiconductor etc. physical hardware implementationtechnologies.

To address the challenges per above, there is a need for inventionsenabling scalable, multi-application dynamic concurrent execution onparallel processing systems, with high resource utilization efficiency,high application processing on-time throughput performance, as wellbuilt-in, architecture based security and reliability.

SUMMARY

An aspect of the invention provides systems and methods for arrangingsecure and reliable, concurrent execution of a set of internallyparallelized and pipelined software programs on a pool of processingresources shared dynamically among the programs, wherein the dynamicsharing of the resources is based at least in part on i) processinginput data loads for instances and tasks of the programs and ii)contractual capacity entitlements of the programs.

An aspect of the invention provides methods and systems for intelligent,destination task defined prioritization of inter-task communications(ITC) for a computer program, for architectural ITC performanceisolation among a set of programs executing concurrently on adynamically shared data processing platform, as well as for prioritizinginstances of the program tasks for execution at least in part based onwhich of the instances have available to them their input data,including ITC data, enabling any given one of such instances to executeat the given time.

An aspect of the invention provides a system for prioritizing instancesof a software program for execution. Such a system comprises: 1) asubsystem for determining which of the instances are ready to execute onan array of processing cores, at least in part based on whether a givenone of the instances has available to it input data to process, and 2) asubsystem for assigning a subset of the instances for execution on thearray of cores based at least in part on the determining. Variousembodiments of that system include further features such as featureswhereby a) the input data is from a data source such that the giveninstance has assigned a high priority for purposes of receiving data; b)the input data is such data that it enables the given program instanceto execute; c) the subset includes cases of none, some as well as all ofthe instances of said program; d) the instance is: a process, a job, atask, a thread, a method, a function, a procedure or an instance any ofthe foregoing, or an independent copy of the given program; and/or e)the system is implemented by hardware logic that is able to operatewithout software involvement.

An aspect of the invention provides a hardware logic implemented methodfor prioritizing instances of a software program for execution, withsuch a method involving: classifying instances of the program into thefollowing classes, listed in the order from higher to lower priority forexecution, i.e., in their reducing execution priority order: (I)instances indicated as having high priority input data for processing,and (II) any other instances. Various embodiments of that method includefurther steps and features such as features whereby a) the otherinstances are further classified into the following sub-classes, listedin their reducing execution priority order: (i) instances indicated asable to execute presently without the high priority input data, and (ii)any remaining instances; b) the high priority input data is data that isfrom a source where its destination instance, of said program, isexpecting high priority input data; c) a given instance of the programcomprises tasks, with one of said tasks referred to as a destinationtask and others as source tasks of the given instance, and for the giveninstance, a unit of the input data is considered high priority if it isfrom such one of the source tasks that the destination task has assigneda high priority for inter-task communications to it; d) for any givenone of the instances, a step of computing a number of its non-emptysource task specific buffers among its input data buffers such thatbelong to source tasks of the given instance indicated at the time ashigh priority source tasks for communications to the destination task ofthe given instance, with this number referred to as an H number for itsinstance, and wherein, within the class I), the instances areprioritized for execution at least in part according to magnitudes oftheir H numbers, in descending order such that an instance with agreater H number is prioritized before an instance with lower H number;e) in case of two or more of the instances tied for the greatest Hnumber, such tied instances are prioritized at least in part accordingto their respective total numbers of non-empty input data buffers,and/or f) at least one of the instances is either a process, a job, atask, a thread, a method, a function, a procedure, or an instance any ofthe foregoing, or an independent copy of the given program.

An aspect of the invention provides a system for processing a set ofcomputer programs instances, with inter-task communications (ITC)performance isolation among the set of program instances. Such a systemcomprises: 1) a number of processing stages; and 2) a group ofmultiplexers connecting ITC data to a given stage among the processingstages, wherein a multiplexer among said group is specific to one givenprogram instance among said set. The system hosts each task of the givenprogram instance at different one of the processing stages, and supportscopies of same task software code being located at more than one of theprocessing stages in parallel. Various embodiments of this systeminclude further features such as a) a feature whereby at least one ofprocessing stages comprises multiple processing cores such as CPUexecution units, with, for any of the cores, at any given time, one ofthe program instances assigned for execution; b) a set of source taskspecific buffers for buffering data destined for a task of the givenprogram instance located at the given stage, referred to as adestination task, and hardware logic for forming a hardware signalindicating whether sending ITC is presently permitted to a given bufferamong the source task specific buffers, with such forming based at leastin part on a fill level of the given buffer, and with such a signalbeing connected to a source task for which the given buffer is specificto; c) a feature providing, for the destination task, a set of sourcetask specific buffers, wherein a given buffer is specific to one of theother tasks of the program instance for buffering ITC from said othertask to the destination task; d) feature wherein the destination taskprovides ITC prioritization information for other tasks of the programinstance located at their respective ones of the stages; d) a featurewhereby the ITC prioritization information is provided by thedestination task via a set of one or more hardware registers, with eachregister of the set specific to one of the other tasks of the programinstance, and with each register configured to store a value specifyinga prioritization level of the task that it is specific to, for purposesof ITC communications to the destination task; e) an arbitratorcontrolling from which source task of the program instance themultiplexer specific to that program instance will read its next ITCdata unit for the destination task; and/or f) a feature whereby thearbitrator prioritizes source tasks of the program instance forselection by the multiplexer to read its next ITC data unit based atleast in part on at least one of: (i) source task specific ITCprioritization information provided by the destination task, and (ii)source task specific availability information of ITC data for thedestination task from the other tasks of the program instance.

Accordingly, aspects of the invention involve application-programinstance specific hardware logic resources for secure and reliable ITCamong tasks of application program instances hosted at processing stagesof a multi-stage parallel processing system. Rather than seeking tointer-connect the individual processing stages or cores of themulti-stage manycore processing system as such, the invented mechanismsefficiently inter-connect the tasks of any given application programinstance using the per application program instance specificinter-processing stage ITC hardware logic resources. Due to the ITCbeing handled with such application program instance specific hardwarelogic resources, the ITC performance experience by one applicationinstance does not depend on the ITC resource usage (e.g. data volume andinter-task communications intensiveness) of the other applicationssharing the given data processing system per the invention. This resultsin effective inter-application isolation for ITC in a multi-stageparallel processing system shared dynamically among multiple applicationprograms.

An aspect of the invention provides systems and methods for schedulinginstances of software programs for execution based at least in part on(1) availability of input data of differing priorities for any given oneof the instances and/or (2) availability, on their fast-access memories,of memory contents needed by any given one of the instances to execute.

An aspect of the invention provides systems and methods for optimallyallocating and assigning input port capacity to a data processingsystems among data streams of multiple software programs based at leastin part on input data load levels and contractual capacity entitlementsof the programs.

An aspect of the invention provides systems and methods for resolutionof resource access contentions, for resources including computing,storage and communication resources such as memories, queues, ports orprocessors. Such methods enable multiple potential user systems for ashared resource, in a coordinated and fair manner, to avoid conflictingresource access decisions, even while multiple user systems are decidingon access to set of shared resources concurrently, including at the sameclock cycle.

An aspect of the invention provides systems and methods for loadbalancing, whereby the load balancer is configured to forward, by itsfirst layer, any packets without destination instance within itsdestination application specified (referred to as no-instance-specifiedpackets or NIS packets for short) it receives from its network input tosuch one of the processing systems in the local load balancing groupthat presently has the highest score for accepting NIS packets for thedestination app of the given NIS packet. The load balancers further havedestination processing system (i.e. for each given application, instancegroup) specific sub-modules, which, for NIS packets forwarded to them bythe first layer balancing logic, specify a destination instance amongthe available, presently inactive instance resources of the destinationapp of a given NIS packet to which to forward the given NIS packet. Inat least some embodiments of the invention, the score for accepting NISpackets for a destination processing system among the load balancinggroup is based at least in part on the amount of presently inactiveinstance resources at the given processing system for the destinationapplication of a given NIS packet.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows, in accordance with an embodiment of the invention, afunctional block diagram for a load balancing architecture for a bank ofprocessor systems, such as those discussed in the following withreference to the remaining FIGS.

FIG. 2 shows, in accordance with an embodiment of the invention, afunctional block diagram for a multi-stage manycore processing systemshared dynamically among a set of software program instances, with thesystem providing capabilities for optimally scheduling inter-taskcommunications (ITC) units between various tasks of any one of theprogram instances, as well as scheduling and placing instances of agiven program task for execution on the processing stages of the system,at least in part based on which of the instances have available for themthe input data, e.g. ITC data, needed by them to execute.

FIG. 3 shows, in accordance with an embodiment of the invention, afunctional block diagram for a receive (RX) logic module of any of theprocessing stages of the multi-stage manycore processor system per FIG.2.

FIG. 4 shows, in accordance with an embodiment of the invention, afunctional block diagram for an application program specific submoduleof the processing stage RX logic module per FIG. 3.

FIG. 5 shows, in accordance with an embodiment of the invention, afunctional block diagram for an application program instance specificsubmodule of the application program specific submodule per FIG. 4.

FIG. 6 shows, in accordance with an embodiment of the invention, afunctional block diagram for logic resources within one of theprocessing stages of a system 1 per FIG. 2 for connecting ITC data frominput buffers of the RX logic (per FIGS. 3-5) to the manycore processorof the local processing stage.

FIG. 7 shows, in accordance with an embodiment of the invention, afunctional block diagram for the application load adaptive manycoreprocessor of a processing stage of the multi-stage processing system perpreceding FIGS.

DETAILED DESCRIPTION

FIGS. and related descriptions in the following provide specificationsfor embodiments and aspects of hardware-logic based systems and methodsfor inter-task communications (ITC) with destination task defined sourcetask prioritization, for input data availability based prioritization ofinstances of a given application task for execution on processing coresof a processing stage hosting the given task, for architecture-basedapplication performance isolation for ITC in multi-stage manycore dataprocessing system, as well as for load balancing of incoming processingdata units among a group of such processing systems.

The invention is described herein in further detail by illustrating thenovel concepts in reference to the drawings. General symbols andnotations used in the drawings:

-   -   Boxes indicate a functional module comprising digital hardware        logic.    -   Arrows indicate a digital signal flow. A signal flow may        comprise one or more parallel bit wires. The direction of an        arrow indicates the direction of primary flow of information        associated with it with regards to discussion of the system        functionality herein, but does not preclude information flow        also in the opposite direction.    -   A dotted line marks a border of a group of drawn elements that        form a logical entity with internal hierarchy.    -   An arrow reaching to a border of a hierarchical module indicate        connectivity of the associated information to/from all        sub-modules of the hierarchical module.    -   Lines or arrows crossing in the drawings are decoupled unless        otherwise marked.    -   For clarity of the drawings, generally present signals for        typical digital logic operation, such as clock signals, or        enable, address and data bit components of write or read access        buses, are not shown in the drawings.

General notes regarding this specification (incl. text in the drawings):

-   -   For brevity: ‘application (program)’ is occasionally written in        as ‘app’, ‘instance’ as ‘inst’ and ‘application-task/instance’        as ‘app-task/inst’ and so forth.    -   Terms software program, application program, application and        program are used interchangeably in this specification, and each        generally refers to any type of executable computer program.    -   In FIG. 5, and through the related discussions, the buffers 260        are considered to be First-in First-Out buffers (FIFO); however        also other types than first-in first-out buffers can be used in        various embodiments.

Illustrative embodiments and aspects of the invention are described inthe following with references to the FIGS.

FIG. 1 presents the load balancing architecture for a row of processingsystems per this description, comprising a set 4 of T load balancers 3and a load balancing group 2 of S processing systems 1 (T and S arepositive integers). Per this architecture, each of the balancers forwardany no-instance-specific (NIS) packets (i.e. packets without a specificinstance of their destination applications identified) arriving to themvia their network inputs to one of the processing systems of the group,based on the NIS packet forwarding preference scores (for thedestination app of the given NIS packet) of the individual processingsystems of the load balancing group 2.

The load balancing per FIG. 1 for a bank 2 of the processing systemsoperates as follows:

-   -   The processing systems 1 count, for each of the application        programs (apps) hosted on them:        -   a number X of their presently inactive instance resources,            i.e., the number of additional parallel instances of the            given app at the given processing system that could be            activated at the time; and        -   from the above number, the portion Y (if any) of the            additional activatable instances within the Core Entitlement            (CE) level of the given app, wherein the CE is a number of            processing cores at (any one of) the processing stages of            the given processing system up to which the app in question            is assured to get its requests for processing cores (to be            assigned for its active instances) met;        -   the difference W=X−Y. The quantities X and/or W and Y, per            each of the apps hosted on the load balancing group 2, are            signaled 5 from each processing system 1 to the load            balancers 4.    -   In addition, load balancing logic 4 computes the collective sum        Z of the Y numbers across all the apps (with this        across-apps-sum Z naturally being the same for all apps on a        given processing system).    -   From the above numbers, for each app, the load balancer module 4        counts a no-instance-specified (NIS) packet forwarding        preference score (NIS score) for each processing system in the        given load balancing group with a formula of: A*Y+B*W+C*Z, where        A, B and C are software programmable, defaulting to e.g. A=4,        B=1 and C=2.        -   In forming the NIS scores for a given app (by formula per            above), a given instance of the app under study is deemed            available for NIS packets at times that the app instance            software has set an associated device register bit (specific            to that app-inst) to an active value, and unavailable            otherwise. The multiplexing (muxing) mechanism used to            connect the app-instance software, from whichever core at            its host manycore processor it may be executing at any given            time, to its app-instance specific memory, is used also for            connecting the app-instance software to its NIS-availability            control device register.        -   The app-instance NIS availability control register of a            given app-instance is reset (when the app-instance software            otherwise would still keep its NIS availability control            register at its active stage) also automatically by            processing stage RX logic hardware whenever there is data at            the input buffer for the given app-instance.    -   Each of the processing systems in the given load balancing group        signals their NIS scores for each app hosted on the load        balancing group to each of the load balancers 4 in front of the        row 2 of processing systems. Also, the processing systems 1        provide to the load balancers app specific vectors (as part of        info flows 9) indicating which of their local instance resources        of the given app are available for receiving NIS packets (i.e.        packets with no destination instance specified).    -   Data packets from the network inputs 10 to the load balancing        group include bits indicating whether any given packet is a NIS        packet such that has its destination app but not any particular        instance of the app specified. The load balancer 3 forwards any        NIS packet it receives from its network input 10 to the        processing system 1 in the local load balancing group 2 with the        highest NIS score for the destination app of the given NIS        packet. (In case of ties among the processing systems for the        NIS score for the given destination app, the logic forwards the        packet to the processing system among such tied systems based on        their ID #, e.g. to the system with lowest ID #.) The forwarding        of a NIS packet to a particular processing system 1 (in the load        balancing group 2 of such systems) is done by this first layer        of load balancing logic by forming packet write enable vectors        where each given bit is a packet write enable bit specific to        the processing system within the given load balancing group of        the same system index # as the given bit in its write enable bit        vector. For example, the processing system ID #2 from a load        balancing group of processing systems of ID #0 through ID #4        takes the bit at index 2 of the packet write enable vectors from        the load balancers of the given group. In a straightforward        scheme, the processing system #K within a given load balancing        group hosts the instance group #K of each of the apps hosted by        this group of the processing systems (where K=0,1, . . . ,max nr        of processing systems in the load balancing group less 1).    -   The load balancers 3 further have destination processing system        1 (i.e. for each given app, instance group) specific submodules,        which, for NIS packets forwarded to them by the first layer        balancing logic (per above), specify a destination instance        among the available (presently inactive) instance resources of        the destination app of a given NIS packet to which to forward        the given NIS packet. In an straightforward scheme, for each        given NIS packet forwarded to it, this instance group specific        load balancing submodule selects, from the at-the-time available        instances of the of the destination app, within the instance        group that the given submodule is specific to, the instance        resource with lowest ID #.    -   For other (not NIS) packets, the load balancer logic 3 simply        forwards a given (non NIS) packet to the processing system 1 in        the load balancing group 2 that hosts, for the destination app        of the given packet, the instance group of the identified        destination instance of the packet.    -   According to the forwarding decision per above bullet points,        the (conceptual, actually distributed per the destination        processing systems) packet switch module 6 filters packets from        the output buses 15 of the load balancers 3 to input buses 19 of        the destination processing systems, so that each given        processing system 1 in the load balancing group 2 receives as        active packet transmissions (marked e.g. by write by write        enable signaling) on its input bus 19, from the packets arriving        from the load balancer inputs 10, those packets that were        indicated as destined to the given system 1 at entry to the load        balancers, as well as the NIS packets that the load balancers of        the set 4 forwarded to that given system 1.    -   Note also that the network inputs 10 to the load balancers, as        well as all the bold data path arrows in the FIGS., may comprise        a number of parallel of (e.g. 10 Gbps) ports.    -   The load balancing logic implements coordination among port        modules of the same balancer, so that any given NIS packet is        forwarded, according to the above destination instance selection        logic, to one of such app-instances that is not, at the time of        the forwarding decision, already being forwarded a packet (incl.        forwarding decisions made at the same clock cycle) by port        modules with higher preference rank (e.g. based on lower port #)        of the same balancer. Note that each processing system supports        receiving packets destined for the same app-instance        concurrently from different load balancers (as explained below).    -   The load balancers 3 support, per each app-inst, a dedicated        input buffer per each of the external input ports (within the        buses 10) to the load balancing group. The system thus supports        multiple packets being received (both via the same load balancer        module 3, as well as across the different load balancer modules        per FIG. 1) simultaneously for the same app-instances via        multiple external input ports. From the load balancer input        buffers, data packets are muxed to the processing systems 1 of        the load balancing group so that the entry stage processor of        each of the multi-stage systems (see FIG. 2) in such group        receives data from the load balancers similarly as the        non-entry-stage processors receive data from the other        processing stages of the given multi-stage processing        system—i.e., in a manner that the entry stage (like the other        stages) will get data per each of its app-instances at most via        one of its input ports per a (virtual) source stage at any given        time; the load balancer modules of the given load balancing        group (FIG. 1) appear thus as virtual source processing stages        to entry stage of the multi-stage processing systems of such        load balancing group. The aforesaid functionality is achieved by        logic at module 4 as detailed below:        -   To eliminate packet drops in cases where packets directed to            same app-inst arrive in a time-overlapping manner through            multiple input ports (within the buses 10) of same balancer            3, destination processing system 1 specific submodules at            modules 3 buffer input data 15 destined for the given            processing system 1 at app-inst specific buffers, and assign            the processing system 1 input ports (within the bus 19            connecting to their associated processing system 1) among            the app-insts so that each app-inst is assigned at any given            time at most one input port per a load balancer 3. (Note            that inputs to a processing system 1 from different load            balancers 3 are handled by the entry stage (FIG. 2) the same            way as the other processing stages 300 handle inputs from            different source stages, as detailed in connection to FIG.            5—in a manner that supports concurrent reception of packets            to the same destination app-inst from multiple source            stages.) More specifically, the port capacity 19 for            transfer of data from load balancers 4 to the given            processing system 1 entry-stage buffers gets assigned using            the same algorithm as is used for assignment of processing            cores between the app-instances at the processing stages            (FIG. 7), i.e., in a realtime input data load adaptive            manner, while honoring the contractual capacity entitlements            and fairness among the apps for actually materialized            demands. This algorithm, which allocates at most one of the            cores per each of the app-insts for the core allocation            periods following each of its runs—and similarly assigns at            most one of the ports at buses 19 to the given processing            system 1 per each of the app-inst specific buffers queuing            data destined for that processing system from any given            source load balancer 3—is specified in detail in [1],            Appendix A, Ch. 5.2.3. By this logic, the entry stage of the            processing system (FIG. 2) will get its input data same way            as the other stages, and there thus is no need to prepare            for cases of multiple packets to same app-inst arriving            simultaneously at any destination processing stage from any            of its source stages or load balancers. This logic also            ensures that any app with moderate input bandwidth            consumption will gets its contractually entitled share of            the processing system input bandwidth (i.e. the logic            protects moderate bandwidth apps from more input data            intensive neighbors).    -   Note that since packet transfer within a load balancing group        (incl. within the sub-modules of the processing systems) is        between app-instance specific buffers, with all the overhead        bits (incl. destination app-instance ID) transferred and        buffered as parallel wires besides the data, core allocation        period (CAP) boundaries will not break the packets while being        transferred from the load balancer buffers to a given processing        system 1 or between the processing stages of a given multi-stage        system 1.

The mechanisms per above three bullet points are designed to eliminateall packet drops in the system such that are avoidable by system design,i.e., for reasons other than app-instance specific buffer overflowscaused be systemic mismatches between input data loads to a givenapp-inst and the capacity entitlement level subscribed to by the givenapp.

FIG. 2 provides, according to an embodiment of the invention, afunctional block diagram for a multistage manycore processor system 1shared dynamically multiple concurrent application programs (apps), withhardware logic implemented capabilities for scheduling tasks ofapplication program instances and prioritizing inter-task communications(ITC) among tasks of a given app instance, based at least in part on,for any given app-inst, at a given time, which tasks are expecting inputdata from which other tasks and which tasks are ready to execute oncores of the multi-stage manycore processing system, with theready-to-execute status of a given task being determined at least inpart based on whether the given task has available to it the input datafrom other tasks or system 1 inputs 19 so as to enable it to execute atthe given time, including producing its processing outputs, such as ITCcommunications 20 to other tasks or program processing results etc.communications for external parties via external outputs 50. Operationand internal structure and elements of FIG. 2, other than for theaspects described herein, is, according to at least some embodiments ofthe invention, per [1], which the reader may review before thisspecification for context and background material.

In the architecture per FIG. 2, the multi-stage manycore processorsystem 1 is shared dynamically among tasks of multiple applicationprograms (apps) and instances (insts) thereof, with, for each of theapps, each task located at one of the (manycore processor) basedprocessing stages 300. Note however that, for any given app-inst, copiesof same task software (i.e. copies of same software code) can be locatedat more than one of the processing stages 300 of a given system 1; thusthe architecture per FIG. 2, with its any-to-any ITC connectivitybetween the stages 300, supports organizing tasks of a program flexiblyfor any desirable mixes or matches of pipelined and/or parallelizedprocessing.

General operation of the application load adaptive, multi-stage paralleldata processing system per FIG. 2, focusing on the main inputs tooutputs data flows, is as follows: The system provides data processingservices to be used by external parties (e.g. by clients of the programshosted on the system) over networks. The system 1 receives data units(e.g. messages, requests, data packets or streams to be processed) fromits users through its inputs 19, and transmits the processing results tothe relevant parties through its network outputs 50. Naturally thenetwork ports of the system of FIG. 2 can be used also for connectingwith other (intermediate) resources and services (e.g. storage,databases etc.) as desired for the system to produce the requestedprocessing results to the relevant external parties.

The application program tasks executing on the entry stage manycoreprocessor are typically of ‘master’ type for parallelized/pipelinedapplications, i.e., they manage and distribute the processing workloadsfor ‘worker’ type tasks running (in pipelined and/or parallel manner) onthe worker stage manycore processing systems (note that the processorsystem hardware is similar across all instances of the processing stages300). The instances of master tasks typically do preliminary processing(e.g. message/request classification, data organization) and workflowmanagement based on given input data units (packets), and then typicallyinvolve appropriate worker tasks at their worker stage processors toperform the data processing called for by the given input packet,potentially in the context of and in connection with other related inputpackets and/or other data elements (e.g. in memory or storage resourcesaccessible by the system) referred to by such packets. (The processorshave access to system memories through interfaces also additional to theIO ports shown in FIG. 2, e.g. as described in [1], Appendix A, Ch.5.4). Accordingly, the master tasks typically pass on the received dataunits (using direct connection techniques to allow most of the datavolumes being transferred to bypass the actual processor cores) throughthe (conceptual) inter-stage packet-switch (PS) to the worker stageprocessors, with the destination application-task instance (and thereby,the destination worker stage) identified for each data unit as describedin the following.

To provide isolation among the different applications configured to runon the processors of the system, by default the hardware controller ofeach processor 300, rather than any application software (executing on agiven processor), inserts the application ID # bits for the data packetspassed to the PS 200. That way, the tasks of any given applicationrunning on the processing stages in a system can trust that the packetsthey receive from the PS are from its own application. Note that thecontroller determines, and therefore knows, the application ID # thateach given core within its processor is assigned to at any given time,via the application-instance to core mapping info that the controllerproduces. Therefore the controller is able to insert thepresently-assigned app ID # bits for the inter-task data units beingsent from the cores of its processing stage over the core-specificoutput ports to the PS.

While the processing of any given application (server program) at asystem per FIG. 2 is normally parallelized and/or pipelined, andinvolves multiple tasks (many of which tasks and instances thereof canexecute concurrently on the manycore arrays of the processing stages300), the system enables external parties to communicate with any suchapplication hosted on the system without knowledge about any specifics(incl. existence, status, location) of their internal tasks orinstances. As such, the incoming data units to the system are expectedto identify just their destination application, and when applicable, theapplication instance. Moreover, the system enables external parties tocommunicate with any given application hosted on a system through any ofthe network input ports 10 of any of the load balancers 3, without suchexternal parties knowing whether or at which cores 520 (FIG. 7) orprocessing stages 300 any instance of the given application task(app-task) may be executing at any time.

Notably, the architecture enables the aforesaid flexibility andefficiency through its hardware logic functionality, so that no systemor application software running on the system needs to either keep trackof whether or where any of the instances of any of the app-tasks may beexecuting at any given time, or which port any given inter-task orexternal communication may have used. Thus the system, while providing ahighly dynamic, application workload adaptive usage of the systemprocessing and communications resources, allows the software running onand/or remotely using the system to be designed with a straightforward,abstracted view of the system: the software (both remote and localprograms) can assume that all the applications, and all their tasks andinstances, hosted on the given system are always executing on theirvirtual dedicated processor cores within the system. Also, where useful,said virtual dedicated processors can also be considered by software tobe time-share slices on a single (unrealistically high speed) processor.

The presented architecture thereby enables achieving, at the same time,both the vital application software development productivity (simple,virtual static view of the actually highly dynamic processing hardware)together with high program runtime performance (scalable concurrentprogram execution with minimized overhead) and resource efficiency(adaptively optimized resource allocation) benefits. Techniques enablingsuch benefits of the architecture are described in the following throughmore detailed technical description of the system 1 and its subsystems.

The any-to-any connectivity among the app-tasks of all the processingstages 300 provided by the PS 200 enables organizing the worker tasks(located at the array of worker stage processors) flexibly to suit theindividual demands (e.g. task inter-dependencies) of any givenapplication program on the system: the worker tasks can be arranged toconduct the work flow for the given application using any desiredcombinations of parallel and pipelined processing. E.g., it is possibleto have the same task of a given application located on any number ofthe worker stages in the architecture per FIG. 2, to provide a desirednumber of parallel copies of a given task per an individual applicationinstance, i.e. to support also data-parallelism, along with taskconcurrency.

The set of applications configured to run on the system can have theirtasks identified by (intra-app) IDs according to their descending orderof relative (time-averaged) workload levels. Under such (intra-app) taskID assignment principle, the sum of the intra-application task IDs, eachrepresenting the workload ranking of its tasks within its application,of the app-tasks hosted at any given processing system is equalized byappropriately configuring the tasks of differing ID #s, i.e. ofdiffering workload levels, across the applications for each processingsystem, to achieve optimal overall load balancing. For instance, in caseof T=4 worker stages, if the system is shared among M=4 applications andeach of that set of applications has four worker tasks, for eachapplication of that set, the busiest task (i.e. the worker task mostoften called for or otherwise causing the heaviest processing load amongtasks of the app) is given task ID #0, the second busiest task ID #1,the third busiest ID #2, and the fourth ID #3. To balance the processingloads across the applications among the worker stages of the system, theworker stage #t gets task ID #t+m (rolling over at 3 to 0) of theapplication ID #m (t=0,1, . . . T−1; m=0,1, . . . M−1) (note that themaster task ID #4 of each app is located at the entry/exit stages). Inthis example scenario of four application streams, four worker tasks perapp as well as four worker stages, the above scheme causes the task IDsof the set of apps to be placed at the processing stages per Table 1below:

TABLE 1 App ID# m (to right) Processing worker stage# t (below) 0 1 2 30 0 1 2 3 1 1 2 3 0 2 2 3 0 1 3 3 0 1 2

As seen in the example of Table 1, the sum of the task ID #s (with eachtask ID # representing the workload ranking of its task within its app)is the same for any row i.e. for each worker stage. This load balancingscheme can be straightforwardly applied for differing numbers ofprocessing stages/tasks and applications, so that the overall taskprocessing load is to be, as much as possible, equal across allworker-stage processors of the system. Advantages of such schemesinclude achieving optimal utilization efficiency of the processingresources and eliminating or at least minimizing the possibility andeffects of any of the worker-stage processors forming system-wideperformance bottlenecks.

A non-exclusive alternative task to stage placement principle targetsgrouping tasks from the apps in order to minimize any variety among theprocessing core types demanded by the set of app-tasks placed on anygiven individual processing stage; that way, if all app-tasks placed ona given processing stage optimally run on the same processing core type,there is no need for reconfiguring the core slots of the manycore arrayat the given stage regardless which of the locally hosted app-tasks getassigned to which of its core slots (see [1], Appendix A, Ch. 5.5 fortask type adaptive core slot reconfiguration, which may be used when theapp-task located on the given processing stage demand differentexecution core types).

FIGS. 3-5 present the processing stage, app, app-instance levelmicroarchitectures for the processing stage receive (RX) logic modules201 (which collectively accomplish the functionality of the conceptualinter-stage packet-switch (PS) module of FIG. 2).

For a system of FIG. 2, note that the functionality of the conceptualinter-stage PS 200 is actually realized by instantiating the logic perFIG. 3 (and its submodules) as the RX logic of each manycore processingsystem 300 (referred to as a stage) in the multi-stage architecture;there is no need for other logic to the PS. Accordingly, in the hardwareimplementation, the stage RX logic 201 per FIG. 3-5 is part of theprocessing stage 300 that it interfaces to; i.e., in an actual hardwareimplementation, there is no PS module as its functionality isdistributed to the individual processing stages.

Besides the division of the app-specific submodules 202 of the stage RXlogic per FIG. 3 further to the array 410 of app-instance specificsub-modules 203, FIG. 4 shows how the app-specific RX logic forms, forpurposes of optimally assigning the processing cores of the localmanycore processor among insts of the apps sharing the system, thefollowing info for the given app:

-   -   Formation of a request for a number of processing cores (Core        Demand Figure, CDF) at the local processing stage by the given        app. The logic forms the CDF for the app based on the number of        instances of the app that presently have (1) input data at their        input buffers (with those buffers located at the instance        specific stage RX logic submodules 203 per FIG. 5) and (2) their        on-chip fast-access memory contents ready for the given instance        to execute without access to the slower-access off-chip        memories. In FIG. 4, (1) and (2) per above are signaled to the        app-specific RX logic module 209 via the info flows 429 and 499        from the app-inst specific modules 203 (FIG. 5) and 800 (FIG.        7), respectively, per each of the insts of the app under study.    -   The priority order of instances of the app for purposes of        selecting such instances for execution on the cores of the local        manycore processor.        The info per the above two bullet points are sent from the RX        logic 202 of each app via the info flow 430 to the controller        540 (FIG. 7) of the local manycore processor 500, for the        controller to assign optimal sets of the app-insts for execution        on the cores 520 of the processor 500.

The app-instance specific RX logic per FIG. 5 performs multiplexing 280ITC packets from the source stage i.e. source task (of a given app-inst)specific First-in First-Out buffers (FIFOs) 260 to the local manycoreprocessor via the input port 290 of that processor dedicated to thegiven app instance.

Note that when considering the case of RX logic of the entry-stageprocessing system of the multi-stage architecture per FIG. 4.1, notethat in FIG. 5 and associated descriptions the notion of sourcestage/task naturally is replaced by the source load balancer, except incase of the ITC 20 from the exit stage to entry-stage, in which case thedata source naturally is the exit stage processing system. However, thesame actual hardware logic is instantiated for each occurrence of theprocessing stages 300 (incl. for the RX logic 201 of each stage) in thismulti-stage architecture, and thus the operation of the stage RX logiccan be fully explained by (as is done in the following) by assuming thatthe processing stage under study is instantiated as a worker or exitstage processing system, such that receives its input data from theother processing stages of the given multi-stage manycore processor,rather than from the load balancers of the given load balancing group,as in the case of the entry-stage processors; the load balancers appearto the entry-stage as virtual processing stages. Accordingly, when theRX logic of the entry stage manycore processor is considered, thereferences to ‘source stage’ are to be understood as actually referringto load balancers, and the references to ITC mean input data 19 to themulti-stage manycore processor system—except in case of the ITC 20 fromthe exit stage, as detailed above and as illustrated in FIG. 2. Withthis caveat, the description of the stage RX logic herein is writtenconsidering the operating context of worker and exit stage processors(with the same hardware logic being used also for the entry-stage).

Before the actual multiplexer, the app-instance specific RX logic perFIG. 5 has a FIFO module 245 per each of the source stages. Thesource-stage specific FIFO module comprises:

-   -   The actual FIFO 260 for queuing packets from its associated        source stage that are destined to the local task of the        app-instance that the given module per FIG. 5 is specific to.    -   A write-side multiplexer 250 (to the above referred FIFO)        that (1) takes as its data inputs 20 the processing core        specific data outputs 210 (see FIG. 7) from the processing stage        that the given source-stage specific FIFO module is specific        to, (2) monitors (via the data input overhead bits identifying        the app-instance and destination task within it for any given        packet transmission) from which one of its input ports 210        (within the bus 20) it may at any given time be receiving a        packet destined to the local task of the app-instance that the        app-instance specific RX logic under study is specific to, with        such an input referred to as the selected input, and (3)        connects 255 to its FIFO queue 260 the packet transmission from        the present selected input. Note that at any of the processing        stages, at any given time, at most one processing core will be        assigned for any given app instance. Thus any of the source        stage specific FIFO modules 245 of the app-instance RX logic per        FIG. 5 can, at any given time, receive data destined to the        local task of the app-instance that the given app-instance RX        logic module is specific to from at most one of the (processing        core specific) data inputs of the write-side multiplexer (mux)        250 of the given FIFO module. Thus there is no need for separate        FIFOs per each of the (e.g. 16 core specific) ports of the data        inputs 20 at these source stage specific FIFO modules, and        instead, just one common FIFO suffices per each given source        stage specific buffering module 245.        For clarity, the “local” task refers to the task of the        app-instance that is located at the processing stage 300 that        the RX logic under study interfaces to, with that processing        stage or processor being referred to as the local processing        stage or processor. Please recall that per any given app, the        individual tasks are located at separate processing stages. Note        though that copies of the same task for a given app can be        located at multiple processing stages in parallel. Note further        that, at any of the processing stages, there can be multiple        parallel instances of any given app executing concurrently, as        well as that copies of the task can be located in parallel at        multiple processing stages of the multi-stage architecture,        allowing for processing speed via parallel execution at        application as well as task levels, besides between the apps.

The app-instance RX module 203 per FIG. 5 further provides arbitratinglogic 270 to decide, at multiplexing packet boundaries 281, from whichof the source stage FIFO modules 245 to mux 280 out the next packet tothe local manycore processor via the processor data input port 290specific to the app-instance under study. This muxing process operatesas follows:

Each given app-instance software provides a logic vector 595 to thearbitrating logic 270 of its associated app-instance RX module 203 suchthat has a priority indicator bit within it per each of its individualsource stage specific FIFO modules 245: while a bit of such a vectorrelating to a particular source stage is at its active state (e.g. logic‘1’), ITC from the source stage in question to the local task of theapp-instance will be considered to be high priority, and otherwisenormal priority, by the arbitrator logic in selecting the source stagespecific FIFO from where to read the next ITC packet to the local(destination) task of the studied app-instance.

The arbitrator selects the source stage specific FIFO 260 (within thearray 240 of the local app-instance RX module 203) for reading 265, 290the next packet per the following source priority ranking algorithm:

-   -   The source priority ranking logic maintains three logic vectors        as follows:        -   1) A bit vector wherein each given bit indicates whether a            source stage of the same index as the given bit is both            assigned by the local (ITC destination) task of the            app-instance under study a high priority for ITC to it and            has its FIFO 260 fill level above a configured monitoring            threshold;        -   2) A bit vector wherein each given bit indicates whether a            source stage of the same index as the given bit is both            assigned a high priority for ITC (to the task of the studied            app-instance located at the local processing stage) and has            its FIFO non-empty;        -   3) A bit vector wherein each given bit indicates whether a            source stage of the same index as the given bit has its FIFO            fill level above the monitoring threshold; and        -   4) A bit vector wherein each given bit indicates whether a            source stage of the same index as the given bit has data            available for reading.    -   The FIFO 260 fill level and data-availability is signaled in        FIG. 5 via info flow 261 per each of the source-stage specific        FIFO modules 245 of the app-inst specific array 240 to the        arbitrator 270 of the app-inst RX module, for the arbitrator,        together with the its source stage prioritization control logic        285, to select 272 the next packet to read from the optimal        source-stage specific FIFO module 245 (as detailed below).    -   The arbitrator logic 270 also forms (by logic OR) an indicator        bit for each of the above vectors 1) through 4) telling whether        the vector associated with the given indicator has any bits in        its active state. From these indicators, the algorithm searches        the first vector, starting from vector 1) and proceeding toward        vector 4), that has one or more active bits; the logic keeps        searching until such a vector is detected.    -   From the detected highest priority ranking vector with active        bit(s), the algorithm scans bits, starting from the index of the        current start-source-stage (and after reaching the max bit index        of the vector, continuing from bit index 0), until it finds a        bit in an active state (logic ‘1’); the index of such found        active bit is the index of the source stage from which the        arbitrator controls its app-instance port mux 280 to read 265        its next ITC packet for the local task of the studied        app-instance.    -   The arbitrator logic uses a revolving (incrementing by one at        each run of the algorithm, and returning to 0 from the maximum        index) starting source stage number as a starting stage in its        search of the next source stage for reading an ITC packet.        When the arbitrator has the appropriate data source (from the        array 240) thus selected for reading 265, 290 the next packet,        the arbitrator 270 directs 272 the mux 280 to connect the        appropriate source-stage specific signal 265 to its output 290,        and accordingly activates, when enabled by the read-enable        control 590 from the app-inst software, the read enable 271        signal for the FIFO 260 of the presently selected source-stage        specific module 245.

Note that the ITC source task prioritization info 595 from the tasksoftware of app-instances to their RX logic modules 203 can changedynamically, as the processing state and demands of input data for agiven app-instance task evolve over time, and the arbitrator modules 270(FIG. 5) apply the current state of the source task prioritization infoprovided to them in selecting from which of the source stages tomultiplex 280 out the next ITC packet over the output port 290 of theapp-instance RX logic. In an embodiment, the local task of a givenapp-inst, when a need arises, writes 575, 595 the respective ITCprioritization levels for its source tasks (of the given app-inst) onits source-task specific ITC prioritization hardware registers, whichare located at (or their info connected to) source-stage prioritizationcontrol logic submodule 285 of the arbitrator 270 of the RX module 203of that given app-inst. Please see FIG. 7 for the muxing 580 of theinput data read control info (incl. source prioritization) from theapp-insts executing at the cores of the array to their associated RXmodules 203.

In addition, the app-instance RX logic per FIG. 5 participates in theinter-stage ITC flow-control operation as follows:

Each of the source stage specific FIFO modules 245 of a givenapp-instance at the RX logic for a given processing stage maintains asignal 212 indicating whether the task (of the app instance under study)located at the source stage that the given FIFO 260 is specific to ispresently permitted to send ITC to the local (destination) task of theapp-instance under study: the logic denies the permit when the FIFO filllevel is above a defined threshold, while it otherwise grants thepermit.

As a result, any given (source) task, when assigned for execution at acore 520 (FIG. 7) at the processing stage where the given task islocated, receives the ITC sending permission signals from each of theother (destination) tasks of its app-instance. Per FIG. 7, these ITCpermissions are connected 213 to the processing cores of the (ITCsource) stages through multiplexers 600, which, according to the control560 from the controller 540 at the given (ITC source) processing stageidentifying the active app-instance for each execution core 520, connect213 the incoming ITC permission signals 212 from the other stages of thegiven multi-stage system 1 to the cores 520 at that stage. For thispurpose, the processing stage provides core specific muxes 600, each ofwhich connects to its associated core the incoming ITC send permitsignals from the ‘remote’ (destination) tasks of the app-instanceassigned at the time to the given core, i.e., from the tasks of thatapp-instance located at the other stages of the given processing system.The (destination) task RX logic modules 203 activate the ITC permissionsignals for times that the source task for which the given permissionsignal is directed to is permitted to send further ITC data to thatdestination task of the given app-inst.

Each given processing stage receive and monitor ITC permit signalsignals 212 from those of the processing stages that the given stageactually is able to send ITC data to; please see FIG. 2 for ITCconnectivity among the processing stages in the herein studiedembodiment of the presented architecture.

The ITC permit signal buses 212 will naturally be connected across themulti-stage system 1 between the app-instance specific modules 203 ofthe RX logic modules 202 of the ITC destination processing stages andthe ITC source processing stages (noting that a given stage 300 will beboth a source and destination for ITC as illustrated in FIG. 2), thoughthe inter-stage connections of the ITC flow control signals are notshown in FIG. 2. The starting and ending points of the of the signalsare shown, in FIG. 5 and FIG. 7 respectively, while the grouping ofthese ITC flow control signals according to which processing stage thegiven signal group is directed to, as well as forming of the stagespecific signal groups according to the app-instance # that any givenITC flow control signal concerns, are illustrated also in FIGS. 3-4. Inconnecting these per app-instance ID # arranged, stage specific groupsof signals (FIG. 3) to any of the processing stages 300 (FIG. 7), theprinciple is that, at arrival to the stage that a given set of suchgroups of signals is directed to, the signals from said groups arere-grouped to form, for each of the app-instances hosted on the system1, a bit vector where a bit of a given index indicates whether the taskof a given app-instance (that the given bit vector is specific to)hosted at this (source) stage under study is permitted at that time tosend ITC data to its task located at the stage ID # of that given index.Thus, each given bit in these bit vectors informs whether the studiedtask of the given app-instance is permitted to send ITC to the task ofthat app-instance with task ID # equal to the index of the given bit.With the incoming ITC flow control signals thus organized toapp-instance specific bit vectors, the above discussed core specificmuxes 600 (FIG. 7) are able to connect to any given core 520 of thelocal manycore array the (task-ID-indexed) ITC flow control bit vectorof the app-instance presently assigned for execution at the given core.By monitoring the destination stage (i.e. destination task) specificbits of the ITC permission bit vector thus connected to the presentexecution core of a task of the studied app-instance located at the ITC(source) processing stage under study (at times that the givenapp-instance actually is assigned for execution), that ITC source taskwill be able to know to which of the other tasks of its app-instancesending ITC is permitted at any given time.

Note that, notwithstanding the functional illustration in FIG. 5, inactual hardware implementation, the FIFO fill-above-thresholdindications from the source stage specific FIFOs 260 of the app-instancespecific submodules of the RX logic modules of the (ITC destination)processing stages of the present multi-stage system are wired directly,though as inverted, as the ITC send permission indication signals to theappropriate muxes 600 of the (ITC source) stages, without going throughthe arbitrator modules (of the app-instance RX logic modules at the ITCdestination stages). Naturally, an ITC permission signal indicating thatthe destination FIFO for the given ITC flow has its fill level presentlyabove the configured threshold is to be understood by the source taskfor that ITC flow as a denial of the ITC permission (until that signalwould turn to indicate that the fill level of the destination FIFO isbelow the configured ITC permission activation threshold).

Each source task applies these ITC send permission signals from a givendestination task of its app-instance at times that it is about to beginsending a new packet over its (assigned execution core specific)processing stage output port 210 to that given destination task. The ITCdestination FIFO 260 monitoring threshold for allowing/disallowingfurther ITC data to be sent to the given destination task (from thesource task that the given FIFO is specific to) is set to a level wherethe FIFO still has room for at least one ITC packet worth of data bytes,with the size of such ITC packets being configurable for a given systemimplementation, and the source tasks are to restrict the remaininglength of their packet transmissions to destination tasks denying theITC permissions according to such configured limits.

The app-level RX logic per FIG. 4 arranges the instances of its app forthe instance execution priority list 535 (sent via info flow 430)according to their descending order of their priority scores computedfor each instance based on their numbers 429 of source stage specificnon-empty FIFOs 260 (FIG. 5) as follows. To describe the forming ofpriority scores, we first define (a non-negative integer) H as thenumber of non-empty FIFOs of the given instance whose associated sourcestage was assigned a high ITC priority (by the local task of the givenapp-instance hosted at the processing stage under study). We also define(a non-negative integer) L as the number of other (non-high ITC prioritysource task) non-empty FIFOs of the given instance. With H and L thusdefined, the intra-app execution priority score P for a given instancespecific module (of the present app under study) is formed withequations as follows, with different embodiments having differingcoefficients for the factors H, L and the number of tasks for the app,T:

-   -   for H>0, P=T−1+2H+L; and    -   for H=0, P=L.

The logic for prioritizing the instances of the given app for itsexecution priority list 535, via a continually repeating process,signals (via hardware wires dedicated for the purpose) to the controller540 of the local manycore processor 500 (FIG. 7) this instance executionpriority list using the following format:

The process periodically starts from priority order 0 (i.e. the app'sinstance with the greatest priority score P), and steps through theremaining priority orders 1 through the maximum supported number ofinstances for the given application (specifically, for its task locatedat the processing stage under study) less 1, producing one instanceentry per each step on the list that is sent to the controller as suchindividual entries. Each entry of such a priority list comprises, as itscore info, simply the instance ID # (as the priority order of any giveninstance is known from the number of clock cycles since the bit pulsemarking the priority order 0 at the start of a new list). To simplifythe logic, also the priority order (i.e. the number of clock cyclessince the bit pulse marking the priority order 0) of any given entry onthese lists is sent along with the instance ID #.

At the beginning of its core to app-instance assignment process, thecontroller 540 of the manycore processor uses the most recent set ofcomplete priority order lists 535 received from the application RXmodules 202 to determine which (highest priority) instances of eachgiven app to assign for execution for the next core allocation period onthat processor.

Per the foregoing, the ITC source prioritization, program instanceexecution prioritization and ITC flow control techniques provideeffective program execution optimization capabilities for each of a setof individual programs configured to dynamically share a given dataprocessing system 1 per this description, without any of the programsimpacting or being impacted by in any manner the other programs of suchset. Moreover, for ITC capabilities, also the individual instances (e.g.different user sessions) of a given program are fully independent fromeach other. The herein described techniques and architecture thusprovide effective performance and runtime isolation between individualprograms among groups of programs running on the dynamically sharedparallel computing hardware.

From here, we continue by exploring the internal structure and operationof a given processing stage 300 beyond its RX logic per FIGS. 3-5, withreferences to FIGS. 6 and 7.

Per FIG. 6, any of the processing stages 300 of the multi-stage system 1per FIG. 2 has, besides the RX logic 201 and the actual manycoreprocessor system (FIG. 7), an input multiplexing subsystem 450, whichconnects input data packets from any of the app-instance specific inputports 290 to any of the processing cores 520 of the processing stage,according to which app-instance is executing at any of the cores at anygiven time.

The monitoring of the buffered input data availability 261 at thedestination app-instance FIFOs 260 of the processing stage RX logicenables optimizing the allocation of processing core capacity of thelocal manycore processor among the application tasks hosted on the givenprocessing stage. Since the controller module 540 of the local manycoreprocessor determines which instances of the locally hosted tasks of theapps in the system 1 execute at which of the cores of the local manycorearray 515, the controller is able to provide the dynamic control 560 forthe muxes 450 per FIG. 6 to connect the appropriate app-instancespecific input data port 290 from the stage RX logic to each of the corespecific input data ports 490 of the manycore array of the localprocessor.

Internal elements and operation of the application load adaptivemanycore processor system 500 are illustrated in FIG. 7. For the intraprocessing stage discussion, it shall be recalled that there is no morethan one task located per processing stage per each of the apps, thoughthere can be up to X (a positive integer) parallel instances of anygiven app-task at its local processing stage (having an array 515 of Xcores). With one task per application per processing stage 300, the termapp-instance in the context of a single processing stage means aninstance of an app-task hosted at the given processing stage understudy.

FIG. 7 provides a functional block diagram for the manycore processorsystem dynamically shared among instances of the locally hostedapp-tasks, with capabilities for application input data load adaptiveallocation of the cores 520 among the applications and for app-instexecution priority based assignment of the cores (per said allocation),as well as for accordantly dynamically reconfigured 550, 560 I/O andmemory access by the app-insts.

As illustrated in FIG. 7, the processor system 500 comprises an array515 of processing cores 520, which are dynamically shared amonginstances of the locally hosted tasks of the application programsconfigured to run on the system 1, under the direction 550, 560 of thehardware logic implemented controller 540. Application program specificlogic functions at the RX module (FIG. 3-5) signal their associatedapplications' capacity demand indicators 430 to the controller. Amongeach of these indicators, the core-demand-figures (CDFs) 530, expresshow many cores their associated app is presently able utilize for its(ready to execute) instances. Each application's capacity demandexpressions 430 for the controller further include a list of its readyinstances in an execution priority order 535.

Any of the cores 520 of a processor per FIG. 7 can comprise any types ofsoftware program and data processing hardware resources, e.g. centralprocessing units (CPUs), graphics processing units (GPUs), digitalsignal processors (DSPs) or application specific processors (ASPs) etc.,and in programmable logic (FPGA) implementation, the core type for anycore slot 520 is furthermore reconfigurable per expressed demands of itsassigned app-task, e.g. per [1], Appendix A, Ch. 5.5.

The hardware logic based controller 540 module within the processorsystem, through a periodic process, allocates and assigns the cores 520of the processor among the set of applications and their instances basedon the applications' core demand figures (CDFs) 530 as well as theircontractual core capacity entitlements (CEs). This application instanceto core assignment process is exercised periodically, e.g. at intervalssuch as once per a defined number (for instance 64, 256 or 1024, or soforth) of processing core clock or instruction cycles. The app-instanceto core assignment algorithms of the controller produce, per theapp-instances on the processor, identification 550 of their executioncores (if any, at any given time), as well as per the cores of thefabric, identification 560 of their respective app-instances to execute.Moreover, the assignments 550, 560 between app-insts and the cores ofthe array 515 control the access between the cores 520 of the fabric andthe app-inst specific memories at the fabric network and memorysubsystem 800 (which can be implemented e.g. per [1] Appendix A, Ch.5.4).

The app-instance to core mapping info 560 also directs the muxing 450 ofinput data from the RX buffers 260 of an appropriate app-instance toeach core of the array 515, as well as the muxing 580 of the input dataread control signals (570 to 590, and 575 to 595) from the core array tothe RX logic submodule (FIG. 5) of the app-instance that is assigned forany given core 520 at any given time.

Similarly, the core to app-inst mapping info 560 also directs the muxing600 of the (source) app-instance specific ITC permit signals (212 to213) from the destination processing stages to the cores 520 of thelocal manycore array, according to which app-instance is presentlymapped to which core.

Further reference specifications for aspects and embodiments of theinvention are in the references [1] through [10].

The functionality of the invented systems and methods described in thisspecification, where not otherwise mentioned, is implemented by hardwarelogic of the system (wherein hardware logic naturally also includes anynecessary signal wiring, memory elements and such).

Generally, this description and drawings are included to illustratearchitecture and operation of practical embodiments of the invention,but are not meant to limit the scope of the invention. For instance,even though the description does specify certain system elements tocertain practical types or values, persons of skill in the art willrealize, in view of this description, that any design utilizing thearchitectural or operational principles of the disclosed systems andmethods, with any set of practical types and values for the systemparameters, is within the scope of the invention. Moreover, the systemelements and process steps, though shown as distinct to clarify theillustration and the description, can in various embodiments be mergedor combined with other elements, or further subdivided and rearranged,etc., without departing from the spirit and scope of the invention.Finally, persons of skill in the art will realize that variousembodiments of the invention can use different nomenclature andterminology to describe the system elements, process phases etc.technical concepts in their respective implementations. Generally, fromthis description many variants and modifications will be understood byone skilled in the art that are yet encompassed by the spirit and scopeof the invention.

1. (canceled)
 2. A data processing system comprising: a plurality ofmanycore processors, each manycore processor including at least oneprogrammable logic core and at least one controller communicativelycoupled to the programmable logic core by an internal data bus andexecuting a control process, the control processes of the plurality ofmanycore processors collectively providing multi-stage processingcontrol, and the programmable logic cores of the plurality of manycoreprocessors collectively providing a multi-stage processing architecturethat implements a plurality of tasks on behalf of a plurality ofapplications; wherein the programmable logic core of each manycoreprocessor collects load data from other programmable logic cores ofother manycore processors and maintains, in a memory of the respectivemanycore processor, a data structure containing the load data indicatinga load of each of a plurality of target programmable logic coresimplementing a designated task of the plurality of tasks of a designatedapplication of the plurality of applications; wherein, when a requestingprogrammable logic core routes a request for the designated task, therequesting programmable logic core routes the request to a given targetprogrammable logic core that is indicated by the load data in the datastructure of the manycore processor of the requesting programmable logiccore to have a lower load than other of the target programmable logiccores; wherein, when the given target programmable logic core receivesthe request from the requesting programmable logic core, the giventarget programmable logic core determines whether a current load of thegiven target programmable logic core is higher than a threshold loadvalue; and based on at least the determination, the given targetprogrammable logic core redirects the request to another programmablelogic core implementing the designated task.
 3. The data processingsystem of claim 2, wherein: the request is a multi-stage request for aplurality of application tasks, wherein the designated task is a nexttask of the plurality of tasks of the designated application; and therequesting programmable logic core receives the multi-stage request fromanother programmable logic core, implements one of the plurality oftasks of the designated application, and routes the multi-stage requestto the target programmable logic core implementing the next task.
 4. Thedata processing system of claim 2, further comprising a plurality ofmanycore processor arrays, each manycore processor array including aninter-stage packet switch, two or more manycore processors of theplurality of manycore processors, and a plurality of portscommunicatively coupling the inter-stage packet switch and the two ormore manycore processors.
 5. The data processing system of claim 2,wherein the request comprises one or more attributes including at leastone of a type of designated task, and packet data of the request.
 6. Amethod implemented by a data processing system, the method comprising:providing a plurality of manycore processors, each manycore processorincluding at least one programmable logic core and at least onecontroller communicatively coupled to the programmable logic core by aninternal data bus and executing a control process, the control processesof the plurality of manycore processors collectively providingmulti-stage processing control, and the programmable logic cores of theplurality of manycore processors collectively providing a multi-stageprocessing architecture that implements a plurality of tasks on behalfof a plurality of applications; at each programmable logic core,collecting load data from other programmable logic cores of othermanycore processors; maintaining, in a memory of the respective manycoreprocessor, a data structure containing the load data indicating a loadof each of a plurality of target programmable logic cores implementing adesignated task of the plurality of tasks of a designated application ofthe plurality of applications; receiving a request for the designatedtask; routing the request to a given target programmable logic core thatis indicated by the load data in the data structure of the manycoreprocessor of the requesting programmable logic core to have a lower loadthan other of the target programmable logic cores; and at the giventarget programmable logic core, receiving the request from therequesting programmable logic core, determining whether a current loadof the given target programmable logic core is higher than a thresholdload value, and based on at least the determination, redirecting therequest to another programmable logic core implementing the designatedtask.
 7. The method of claim 6, wherein: the request is a multi-stagerequest for a plurality of application tasks, wherein the designatedtask is a next task of the plurality of tasks of the designatedapplication; and the method further comprises, at one of the pluralityof manycore processors, receiving the multi-stage request from anotherprogrammable logic core, implementing one of the plurality of tasks ofthe designated application, and routing the multi-stage request to thetarget programmable logic core implementing the next task.
 8. The methodof claim 6, wherein providing the plurality of manycore processorsincludes providing a plurality of manycore processor arrays, eachmanycore processor array including an inter-stage packet switch, two ormore manycore processors of the plurality of manycore processors, and aplurality of ports communicatively coupling the inter-stage packetswitch and the two or more manycore processors.
 9. The method of claim6, wherein the request comprises one or more attributes including atleast one of a type of designated task, and packet data of the request.10. A data processing system comprising: a plurality of manycoreprocessor arrays of a plurality of manycore processor, each manycoreprocessor array including an inter-stage packet switch and two or moremanycore processors of the plurality of manycore processors, eachmanycore processor including at least one programmable logic core and atleast one controller communicatively coupled to the programmable logiccore by an internal data bus and executing a control process, thecontrol processes of the plurality of manycore processors collectivelyproviding multi-stage processing control, and the programmable logiccores of the plurality of manycore processors collectively providing amulti-stage processing architecture that implements a plurality of taskson behalf of a plurality of applications; wherein each programmablelogic core in a manycore processor array of the plurality of manycoreprocessor arrays implements a same application of the plurality ofapplications; wherein each programmable logic core collects load datafrom other programmable logic cores of other manycore processors andmaintains, in a memory of the respective manycore processor, a datastructure containing the load data indicating a load of each of a set oftarget programmable logic cores in a same manycore processor array ofthe plurality of manycore processor arrays; and wherein, when areceiving programmable logic core in a manycore processor array of theplurality of manycore processor arrays receives a request from arequesting programmable logic core, the receiving programmable logiccore determines whether a current load of the receiving programmablelogic core is higher than a threshold load value, and based on at leastthe determination, the receiving programmable logic core redirects therequest to another programmable logic core in the manycore processorarray which the load data indicates has a lower load than otherprogrammable logic cores in the manycore processor array.
 11. The dataprocessing system of claim 10, wherein: the request is a multi-stagerequest for a plurality of application tasks, wherein the designatedtask is a next task of the plurality of tasks of the designatedapplication; and the requesting programmable logic core receives themulti-stage request from another programmable logic core, implements oneof the plurality of tasks of the designated application, and routes themulti-stage request to the target programmable logic core implementingthe next task.
 12. The data processing system of claim 10, wherein therequest comprises one or more attributes including at least one of atype of designated task, and packet data of the request.